Updating DMOD Install Process and Docs #456

robertbartel · 2023-10-31T16:41:06Z

Updating the INSTALL.md file and other documentation related to setting up DMOD
Updating scripts and config file templates to reduce the (required) manual steps to get DMOD up and running
Adding new helper scripts, template files, and template file content in service of the above
A few miscellaneous fixes to issues within the setup process (e.g., Cython version for t-route in ngen image build)

christophertubbs · 2023-10-31T16:51:37Z

doc/DEPENDENCIES.md

+| Dependency                                | Constraints                                 |                                 Notes                                  |
+|-------------------------------------------|:--------------------------------------------|:----------------------------------------------------------------------:|
+| [Usage Dependencies](#usage-dependencies) | Same as above                               |                                                                        |
+| Python                                    | \>=3.8.x                                    |                                                                        |


Might want to bump that up to 3.9.x - we've got 3.9 somewhere in the pipeline (don't remember where). It's probably the github action.

The (perhaps unofficial) rule of thumb has been for DMOD to support all the versions of Python that are currently, officially supported, which includes 3.8 for now. Given ecosystems like RHEL's that aren't as quick to move to new versions, I think this could make a difference in community adoption (which this particular PR is also geared toward).

I could have sworn we had a matrix in the Github Actions to support testing on a handful of different Python versions, including 3.8 and 3.9. Maybe I'm mixing that up with somewhere else and we should add it here.

Regardless, if there is something specific for which we need to consider breaking with the current approach, then let's open a separate issue and formally consider it.

I could have sworn we had a matrix in the Github Actions to support testing on a handful of different Python versions, including 3.8 and 3.9. Maybe I'm mixing that up with somewhere else and we should add it here.

Ah, looks like I was remembering what Nels had set up in the ngen-cal project. We probably should adopt something similar.

christophertubbs · 2023-10-31T18:22:04Z

Is this supposed to have all the DMOD client work?

christophertubbs · 2023-10-31T18:30:19Z

All looks solid - I'll approve if it turns out that the client code wasn't an accidental branch leak.

robertbartel · 2023-10-31T18:37:38Z

@christophertubbs , hmm, yeah that's probably a leak. Let me figure out what I just did.

Update example.env for correct wrf_hydro and troute branches.

Adding commented out examples for absent items and commenting out examples for items that need to be handled with defaults.

Picking subnets for configured Docker networks that are less likely to be encountered in the real world

Adding flag and option to helper script to init all standard service self-signed SSL certs for a deployment.

Documenting several additional required dependencies: - MinIO CLI client - Pip - Python dev headers/libs - A C++ compiler

- docker_pre_deploy_bind_mount_dir_check - docker_pre_deploy_create_secrets - docker_pre_deploy_scheduler_resources

Temporarily deactivate subset service until things can be address (and it is clear that it is necessary).

Updating script to automatically create necessary SSH keys built into job worker images.

Update control_stack.sh to use single node if deploying when there is only one available in the Docker Swarm.

Updating to support a couple different default command names, accept a specified command name for minio client, avoid DNS issues with 'localhost' and avoid SSL issues.

Add script to simplify creating scheduler resource config.

Automating pre-deploy checks/gen of Docker secret files, configured bind mount directories, and scheduler configuration files.

robertbartel · 2023-10-31T18:47:28Z

@christophertubbs, good catch. Thanks for that. I'm not exactly sure how I did that, but I've removed the superfluous commits now.

christophertubbs · 2023-10-31T18:51:18Z

If I had a nickel for every time I had a branch leak I'd have enough money for a pack of post-its to remind me not to cause a branch leak.

christophertubbs

Everything looks great

aaraney · 2023-11-02T14:03:29Z

INSTALL.md

+### Different Python Executable Names
+It is quite common for the Python executables to be named `python3`, `python311`, `pip3`, etc. in the *global* Python environment, instead of just `python` and `pip`.  This is accounted for when [creating a virtual environment](#using-a-virtual-environment), provided the right Python command was used to create it.  Users not doing this should consider setting up shell aliases, symlinks in their path, or some other mechanism to ensure `python` and `pip` execute the desired versions appropriately.
+
+> [!NOTE]


Neat! I didnt know about these markdown directives.

aaraney · 2023-11-02T14:35:44Z

data/example_image_and_domain.yaml

+
+ngen:
+  version:
+    1: '127.0.0.1:5000/ngen:latest'


It is probably worth tracking that both ngen and nwm image names these are hard coded in service.py at the moment. Im sure we will refactor this in the future to be more flexible.

robertbartel added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request maas MaaS Workstream labels Oct 31, 2023

robertbartel requested review from hellkite500, christophertubbs and aaraney October 31, 2023 16:41

christophertubbs reviewed Oct 31, 2023

View reviewed changes

robertbartel added 19 commits October 31, 2023 14:43

Add example image and domain config template.

4a8cb63

Update example.env for correct external branches.

a3ccaa4

Update example.env for correct wrf_hydro and troute branches.

Correcting example.env commented-out examples

73178ad

Adding commented out examples for absent items and commenting out examples for items that need to be handled with defaults.

More unique network selections in example.env.

1890799

Picking subnets for configured Docker networks that are less likely to be encountered in the real world

Remove service-specific container SSL from example.env.

0fdd2ba

Fix several GUI and Redis defaults in example.env.

801b010

Expand gen_cert.sh w/ opt to init std certs.

efcb20a

Adding flag and option to helper script to init all standard service self-signed SSL certs for a deployment.

Updat DEPENDENCIES.md on MinIO CLI, Py packaging.

1b2ccb1

Documenting several additional required dependencies: - MinIO CLI client - Pip - Python dev headers/libs - A C++ compiler

Add missing network opts to example.env.

6bdc7d2

Add more Docker-related shared script funcs.

dd67e72

- docker_pre_deploy_bind_mount_dir_check - docker_pre_deploy_create_secrets - docker_pre_deploy_scheduler_resources

Fix ngen image t-route build Cython version issue.

650a121

Fix old nginx and SSL dir spec in GUI stack config.

b56f23c

Fix service SSL dir names in main stack config.

87deb09

Comment out subset-service from deployment.

33bc25a

Temporarily deactivate subset service until things can be address (and it is clear that it is necessary).

Improve network abilities of control_stack.sh.

5d59303

Update control_stack.sh to gen SSH keys.

f0001d3

Updating script to automatically create necessary SSH keys built into job worker images.

Fix logic for picking obj store deploy config.

78e99f5

Update control_stack.sh to use single node if deploying when there is only one available in the Docker Swarm.

Updates to minio_init.sh.

fa82eac

Updating to support a couple different default command names, accept a specified command name for minio client, avoid DNS issues with 'localhost' and avoid SSL issues.

Add helper script to create simple .env config.

075f0ce

robertbartel added 4 commits October 31, 2023 14:43

Add helper script to create resources.yaml.

0ded5f9

Add script to simplify creating scheduler resource config.

Update control_stack.sh to include more automation.

0d77d07

Automating pre-deploy checks/gen of Docker secret files, configured bind mount directories, and scheduler configuration files.

Updating INSTALL.md with missing and new details.

30bc55d

Updating .gitignore for some defaults in setup.

07428be

robertbartel force-pushed the f/update_install_docs/main branch from 2954c80 to 07428be Compare October 31, 2023 18:45

christophertubbs approved these changes Oct 31, 2023

View reviewed changes

christophertubbs merged commit 776a28c into NOAA-OWP:master Oct 31, 2023
1 check failed

robertbartel deleted the f/update_install_docs/main branch October 31, 2023 20:08

aaraney reviewed Nov 2, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating DMOD Install Process and Docs #456

Updating DMOD Install Process and Docs #456

robertbartel commented Oct 31, 2023

christophertubbs Oct 31, 2023

robertbartel Oct 31, 2023

robertbartel Oct 31, 2023

christophertubbs commented Oct 31, 2023

christophertubbs commented Oct 31, 2023

robertbartel commented Oct 31, 2023

robertbartel commented Oct 31, 2023

christophertubbs commented Oct 31, 2023

christophertubbs left a comment

aaraney Nov 2, 2023

aaraney Nov 2, 2023

Updating DMOD Install Process and Docs #456

Updating DMOD Install Process and Docs #456

Conversation

robertbartel commented Oct 31, 2023

christophertubbs Oct 31, 2023

Choose a reason for hiding this comment

robertbartel Oct 31, 2023

Choose a reason for hiding this comment

robertbartel Oct 31, 2023

Choose a reason for hiding this comment

christophertubbs commented Oct 31, 2023

christophertubbs commented Oct 31, 2023

robertbartel commented Oct 31, 2023

robertbartel commented Oct 31, 2023

christophertubbs commented Oct 31, 2023

christophertubbs left a comment

Choose a reason for hiding this comment

aaraney Nov 2, 2023

Choose a reason for hiding this comment

aaraney Nov 2, 2023

Choose a reason for hiding this comment